[CLOUDP-352109] Run MCK E2E tests against OCI published helm chart #509

viveksinghggits · 2025-10-08T15:48:31Z

Summary

As part of our work to move towards OCI compatible container registries for our helm chart, we are also planning to run our E2E tests against the helm chart that we publish to OCI. This will make sure that we are testing in our E2E what we are providing to our customers.
As part of this effort we have already raised a PR that starts publishing our helm chart to the OCI container registry during dev/staging workflows.
This PR goes and changes our E2E tests to start consuming the helm chart from OCI registry instead of the local helm chart repo.

Additional unrelated change is to fix kubectl-mongodb location that is used for gke code snippets. The fix is to add missing workdir path suffix.

Proof of Work

Successful CI on this PR.
I also ran the test e2e_replica_set_migration locally and it was successful. The logs are here.

publish_helm_chart example output:

[2025/11/05 22:11:19.857] Running command 'subprocess.exec' in function 'publish_helm_chart' (step 7 of 7).
[2025/11/05 22:11:19.919] INFO     2025-11-05 21:11:19,919 [publish_helm_chart]  Packaging chart: mongodb-kubernetes with Version: 0.0.0+690bbc0f836fbf0007154375
[2025/11/05 22:11:19.959] INFO     2025-11-05 21:11:19,959 [publish_helm_chart]  Successfully executed: helm package --version 0.0.0+690bbc0f836fbf0007154375 helm_chart
[2025/11/05 22:11:19.959] INFO     2025-11-05 21:11:19,959 [publish_helm_chart]  Successfully packaged chart and saved it to: /data/mci/55da5d1be5488b41a9ec4e7d02041a86/src/github.com/mongodb/mongodb-kubernetes/mongodb-kubernetes-0.0.0+690bbc0f836fbf0007154375.tgz
[2025/11/05 22:11:19.959] INFO     2025-11-05 21:11:19,959 [publish_helm_chart]  Determined OCI Registry: oci://268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb/helm-charts
[2025/11/05 22:11:19.959] INFO     2025-11-05 21:11:19,959 [publish_helm_chart]  Pushing chart to registry: oci://268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb/helm-charts
[2025/11/05 22:11:20.848] INFO     2025-11-05 21:11:20,848 [publish_helm_chart]  Successfully executed: helm push mongodb-kubernetes-0.0.0+690bbc0f836fbf0007154375.tgz oci://268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb/helm-charts
[2025/11/05 22:11:20.848] INFO     2025-11-05 21:11:20,848 [publish_helm_chart]  Helm Chart mongodb-kubernetes:0.0.0+690bbc0f836fbf0007154375 was published successfully!
[2025/11/05 22:11:20.858] Finished command 'subprocess.exec' in function 'publish_helm_chart' (step 7 of 7) in 1.001053371s.

Downloading the chart with helm pull oci://268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb/helm-charts/mongodb-kubernetes --untar --version 0.0.0+690bbc0f836fbf0007154375 you can see that the Chart.yaml version was properly updated:

apiVersion: v2
description: 'MongoDB Controllers for Kubernetes translate the human knowledge of
  creating a MongoDB instance into a scalable, repeatable, and standardized method. '
home: https://github.com/mongodb/mongodb-kubernetes
icon: https://mongodb-images-new.s3.eu-west-1.amazonaws.com/leaf-green-dark.png
keywords:
  - mongodb
  - database
  - nosql
kubeVersion: '>=1.16-0'
maintainers:
  - email: [email protected]
    name: MongoDB
name: mongodb-kubernetes
type: application
version: 0.0.0+690bbc0f836fbf0007154375

Also run release test and e2e smoke tests are also passing (ignore ibm_power, they are fixed in #573)

Checklist

Have you linked a jira ticket and/or is the ticket in the title?
Have you checked whether your jira ticket required DOCSP changes?
Have you added changelog file?
- use skip-changelog label if not needed
- refer to Changelog files and Release Notes section in CONTRIBUTING.md for more details

github-actions · 2025-10-17T12:20:30Z

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.6.0 Release Notes

New Features

MongoDBCommunity: Added support to configure custom cluster domain via newly introduced spec.clusterDomain resource field. If spec.clusterDomain is not set, environment variable CLUSTER_DOMAIN is used as cluster domain. If the environment variable CLUSTER_DOMAIN is also not set, operator falls back to cluster.local as default cluster domain.
Helm Chart: Introduced two new helm fields operator.podSecurityContext and operator.securityContext that can be used to configure securityContext for Operator deployment through Helm Chart.
MongoDBSearch:
- Switched to gRPC and mTLS for internal communication between mongod and mongot.
  - Since MCK 1.4 the mongod and mongot processess communicated using the MongoDB Wire Protocol and used keyfile authentication. This release switches that to gRPC with mTLS authentication. gRPC will allow for load-balancing search queries against multiple mongot processes in the future, and mTLS decouples the internal cluster authentication mode and credentials among mongod processes from the connection to the mongot process. The Operator will automatically enable gRPC for existing and new workloads, and will enable mTLS authentication if both Database Server and MongoDBSearch resource are configured for TLS.
- Exposed configuration settings for mongot's prometheus metrics endpoint.
  - By default, if spec.prometheus field is not provided then metrics endpoint in mongot is disabled. This is a breaking change. Previously the metrics endpoing was always enabled on port 9946.
  - To enable prometheus metrics endpoint specify empty spec.prometheus: field. It will enable metrics endpoint on a default port (9946). To change the port, set it in spec.prometheus.port field.
- Simplified MongoDB Search setup: Removed the custom Search Coordinator polyfill (a piece of compatibility code previously needed to add the required permissions), as MongoDB 8.2.0 and later now include the necessary permissions via the built-in searchCoordinator role.
- Updated the default mongodb/mongodb-search image version to 0.55.0. This is the version MCK uses if .spec.version is not specified.
- MongoDB deployments using X509 internal cluster authentication are now supported. Previously MongoDB Search required SCRAM authentication among members of a MongoDB replica set. Note: SCRAM client authentication is still required, this change merely relaxes the requirements on internal cluster authentication.

Bug Fixes

Fixed parsing of the customEnvVars Helm value when values contain = characters.
ReplicaSet: Blocked disabling TLS and changing member count simultaneously. These operations must now be applied separately to prevent configuration inconsistencies.
MongoDBSearch now records the reconciled mongot version in status and exposes it via a dedicated kubectl print column.
Fixed inability to specify cluster-wide privileges in custom roles.

Other Changes

kubectl-mongodb plugin: cosign, the signing tool that is used to sign kubectl-mongodb plugin binaries, has been updated to version 3.0.2. With this change, released binaries will be bundled with .bundle files containing both signature and certificate information. For more information on how to verify signatures using new cosign version please refer to -> https://github.com/sigstore/cosign/blob/v3.0.2/doc/cosign_verify-blob.md

1. Move file from scripts to scripts/relese 2. call python files from .sh files so that we can use build_scenario env var

docker/mongodb-kubernetes-tests/Dockerfile

docker/mongodb-kubernetes-tests/kubetester/helm.py

# Conflicts: # .evergreen.yml

# Conflicts: # docker/mongodb-kubernetes-tests/kubetester/helm.py # docker/mongodb-kubernetes-tests/kubetester/operator.py # docker/mongodb-kubernetes-tests/tests/conftest.py

docker/mongodb-kubernetes-tests/kubetester/helm.py

mircea-cosbuc

LGTM

lsierant · 2025-11-12T08:57:56Z

docker/mongodb-kubernetes-tests/Dockerfile


-RUN python3 -m venv /venv && . /venv/bin/activate && pip install --upgrade pip && GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 pip install -r requirements.txt
+# install aws, required to run helm registry login while running the tests
+RUN python3 -m venv /venv  \


we have this already setup on the host - why we need to aws login again in pytest?

Because in pytests we are installing operator using helm charts and for those we need AWS/Quay credentials

and it uses

"--set" "aws.accessKey=${AWS_ACCESS_KEY_ID}" "--set" "aws.secretAccessKey=${AWS_SECRET_ACCESS_KEY}"

that we already set in the test pod?

Will this be also available locally?

AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are stored in private-context and it will be used in locally run tests

lsierant · 2025-11-12T08:59:40Z

scripts/evergreen/e2e/single_e2e.sh

    BUILD_ID="${BUILD_ID:-default_build_id}"
    BUILD_VARIANT="${BUILD_VARIANT:-default_build_variant}"

+    if ! chart_info=$(scripts/dev/run_python.sh scripts/release/oci_chart_info.py --build-scenario "${BUILD_SCENARIO}" 2>&1); then


if we have all of this executed on the host already, why to propagate it to the test pod at all? Couldn't we decide here which helm path to use and use a single HELM_CHART_PATH env var for that that we will use in tests?

This is required for:

helm_registry_login_to_ecr(registry, region) where we need separate value rather than full path.

using version during helm install if the OCI helm charts are used

lsierant · 2025-11-12T09:01:19Z

docker/mongodb-kubernetes-tests/kubetester/helm.py

+    # these are imported here to resolve import cycle issue
+    from tests.conftest import LOCAL_HELM_CHART_DIR, local_operator
+
+    if local_operator():


this is not sufficient - we can run the operator in IDE or in the pod. Also we might want to use oci helm chart even if running the operator locally.

Could we have one env var that we could just point to a proper helm path that we could control in the context and not in pytest?

You've just reinforced my similar concern and I'm fixing it now. I'll add new env var and set default to local helm charts in local-defaults-context

Please take a look again, I've added DEFAULT_HELM_CHART_PATH env var

viveksinghggits requested a review from a team as a code owner October 8, 2025 15:48

viveksinghggits requested review from MaciejKaras and fealebenpae and removed request for a team October 8, 2025 15:48

viveksinghggits marked this pull request as draft October 8, 2025 15:48

viveksinghggits added the skip-changelog Use this label in Pull Request to not require new changelog entry file label Oct 8, 2025

viveksinghggits force-pushed the publish-chart-oci-prpatches branch from 9bd1970 to 5f0938b Compare October 8, 2025 15:50

viveksinghggits force-pushed the e2e-tests-on-published-chart branch from 9d6699b to 4ad79ae Compare October 8, 2025 15:51

viveksinghggits force-pushed the publish-chart-oci-prpatches branch from 5f0938b to 1b61956 Compare October 10, 2025 14:28

viveksinghggits force-pushed the e2e-tests-on-published-chart branch from 4ad79ae to 9394e7c Compare October 10, 2025 14:44

MaciejKaras removed their request for review October 10, 2025 14:58

viveksinghggits force-pushed the publish-chart-oci-prpatches branch from e885549 to 92d2fe6 Compare October 14, 2025 08:22

viveksinghggits force-pushed the e2e-tests-on-published-chart branch 2 times, most recently from 464fc81 to 5d55e9f Compare October 15, 2025 14:43

viveksinghggits changed the title ~~E2e tests on published chart~~ Run MCK E2E tests against OCI published helm chart instead of local chart Oct 17, 2025

viveksinghggits changed the title ~~Run MCK E2E tests against OCI published helm chart instead of local chart~~ [CLOUDP-352109] Run MCK E2E tests against OCI published helm chart Oct 17, 2025

Base automatically changed from publish-chart-oci-prpatches to master October 17, 2025 11:34

viveksinghggits force-pushed the e2e-tests-on-published-chart branch from b485e0c to e81238f Compare October 17, 2025 12:19

viveksinghggits marked this pull request as ready for review October 17, 2025 13:48

viveksinghggits requested review from MaciejKaras and lsierant October 17, 2025 13:48

viveksinghggits added 8 commits October 21, 2025 20:20

Publish helm chart to OCI registry for PR patches

dfbc534

Consume build_scenario via build_info for registry repo and other info

982c77d

Run precommit

0daaf61

Address review comments

4c753fc

1. Move file from scripts to scripts/relese 2. call python files from .sh files so that we can use build_scenario env var

Publish helm chart to OCI registry for PR patches

8b511e5

Install published helm chart instead of local one, during E2E

4f50441

Don't copy entire helm_chart dir, just copy CRDs

087e06a

Delete not needed file

cb2a8d2

Merge branch 'master' into e2e-tests-on-published-chart

8f50012

MaciejKaras reviewed Oct 28, 2025

View reviewed changes

docker/mongodb-kubernetes-tests/Dockerfile Outdated Show resolved Hide resolved

Review fixes

1068518

semgrep-code-mongodb bot reviewed Oct 28, 2025

View reviewed changes

docker/mongodb-kubernetes-tests/kubetester/helm.py Show resolved Hide resolved

MaciejKaras added 7 commits October 28, 2025 15:10

Fix tests

829393f

Merge branch 'master' into e2e-tests-on-published-chart

3aa3fce

Fixes after merge with master

ca7881c

Add version_prefix to build_info.json + other fixes

084b8f2

Fixes

e1700ca

Fix prefix concatenation

2cc7bbd

Merge branch 'master' into e2e-tests-on-published-chart

f53071f

MaciejKaras force-pushed the e2e-tests-on-published-chart branch from d32d374 to f53071f Compare October 30, 2025 11:28

Fix publish_helm_chart task

c482251

MaciejKaras force-pushed the e2e-tests-on-published-chart branch from 5bdab30 to c482251 Compare October 30, 2025 12:57

Fix multicluster tests

1ee164a

MaciejKaras self-assigned this Nov 5, 2025

Merge branch 'master' into e2e-tests-on-published-chart

ffceb0c

# Conflicts: # .evergreen.yml

MaciejKaras force-pushed the e2e-tests-on-published-chart branch from c3fd0b0 to ffceb0c Compare November 5, 2025 21:05

MaciejKaras added 2 commits November 6, 2025 13:15

Use helm charts for release e2e smoke tests

cfee047

Merge branch 'master' into e2e-tests-on-published-chart

1c8f00e

# Conflicts: # docker/mongodb-kubernetes-tests/kubetester/helm.py # docker/mongodb-kubernetes-tests/kubetester/operator.py # docker/mongodb-kubernetes-tests/tests/conftest.py

fealebenpae removed their request for review November 7, 2025 13:53

mircea-cosbuc reviewed Nov 7, 2025

View reviewed changes

docker/mongodb-kubernetes-tests/kubetester/helm.py Outdated Show resolved Hide resolved

docker/mongodb-kubernetes-tests/kubetester/helm.py Outdated Show resolved Hide resolved

MaciejKaras requested a review from mircea-cosbuc November 12, 2025 08:09

mircea-cosbuc approved these changes Nov 12, 2025

View reviewed changes

lsierant reviewed Nov 12, 2025

View reviewed changes

MaciejKaras added 2 commits November 12, 2025 14:05

make review changes

91a6b65

Add DEFAULT_HELM_CHART_PATH env var

2cd6a3d

MaciejKaras force-pushed the e2e-tests-on-published-chart branch from 14bb2a0 to 2cd6a3d Compare November 12, 2025 15:47

[CLOUDP-352109] Run MCK E2E tests against OCI published helm chart #509

Are you sure you want to change the base?

[CLOUDP-352109] Run MCK E2E tests against OCI published helm chart #509

Conversation

viveksinghggits commented Oct 8, 2025 • edited by MaciejKaras Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Proof of Work

Checklist

Uh oh!

github-actions bot commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MCK 1.6.0 Release Notes

New Features

Bug Fixes

Other Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mircea-cosbuc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaciejKaras Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

viveksinghggits commented Oct 8, 2025 •

edited by MaciejKaras

Loading

github-actions bot commented Oct 17, 2025 •

edited

Loading

MaciejKaras Nov 12, 2025 •

edited

Loading